[Spec][MOE][Internal Op] Specification of MOE internal operation #32255
+152
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mitruska
requested review from
maxnick,
mmikolajcz,
riverlijunjie,
rkazants and
yeonbok
mitruska
commented
Oct 1, 2025
Comment on lines
+63
to
+68
| # Experts computation part (GEMM3_SWIGLU) | ||
| x_proj = matmul(reshaped_hidden_states, weight_0, transpose_a=False, transpose_b=True) | ||
| x_proj2 = matmul(reshaped_hidden_states, weight_1, transpose_a=False, transpose_b=True) | ||
| swiglu = swish(x_proj, beta=expert_beta) | ||
| x_proj = x_proj2 * swiglu | ||
| down_proj = matmul(x_proj, weight_2, transpose_a=False, transpose_b=True) |
There was a problem hiding this comment.
GPU plugin request is to transpose those weights at conversion stage, so the MatMul both transpose_a/b attrs should be False at this point:
Suggested change
| # Experts computation part (GEMM3_SWIGLU) | |
| x_proj = matmul(reshaped_hidden_states, weight_0, transpose_a=False, transpose_b=True) | |
| x_proj2 = matmul(reshaped_hidden_states, weight_1, transpose_a=False, transpose_b=True) | |
| swiglu = swish(x_proj, beta=expert_beta) | |
| x_proj = x_proj2 * swiglu | |
| down_proj = matmul(x_proj, weight_2, transpose_a=False, transpose_b=True) | |
| # Experts computation part (GEMM3_SWIGLU) | |
| x_proj = matmul(reshaped_hidden_states, weight_0, transpose_a=False, transpose_b=False) | |
| x_proj2 = matmul(reshaped_hidden_states, weight_1, transpose_a=False, transpose_b=False) | |
| swiglu = swish(x_proj, beta=expert_beta) | |
| x_proj = x_proj2 * swiglu | |
| down_proj = matmul(x_proj, weight_2, transpose_a=False, transpose_b=False) |
cc: @yeonbok
…ts/operation-specs/internal/moe.rst
rkazants
reviewed
Oct 2, 2025
| @@ -0,0 +1,151 @@ | |||
| .. {#openvino_docs_ops_internal_MOE} | |||
|
|
|||
| MOE | |||
There was a problem hiding this comment.
let us not use MoE name because we can use it for external operation and for real MoE operation. Now it is a sort of FusedExperts.
There was a problem hiding this comment.
The routing weights and indices are provided as inputs, so the core MOE idea is preserved, final multiplication and ReduceSum are included.
I would keep the name as is, to make current purpose clear.
The MOE internal op can be refactored as needed in the future, also possibly extended with Router.
rkazants
reviewed
Oct 2, 2025
| .. code-block:: py | ||
| :force: | ||
|
|
||
| # Common part: Reshape hidden states and prepare for expert computation |
There was a problem hiding this comment.
I propose to add router_topk_output_indices into this logic. It will show how weights are prepared. Now it is not clear how router_topk_output_indices is used in the specified operation.
rkazants
reviewed
Oct 2, 2025
tsavina
reviewed
Oct 2, 2025
...articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal/moe.rst
Outdated
Show resolved
Hide resolved
...articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal/moe.rst
Outdated
Show resolved
Hide resolved
...articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal/moe.rst
Outdated
Show resolved
Hide resolved
...articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal/moe.rst
Show resolved
Hide resolved
...articles_en/documentation/openvino-ir-format/operation-sets/operation-specs/internal/moe.rst
Outdated
Show resolved
Hide resolved
Co-authored-by: Tatiana Savina <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
Oct 16, 2025
… experts into MOE (#32183) ### Details: This transformation is for compile time and is not enabled by default, it should be enabled in each plugin with MOE plugin support. Example registration of the fusion transformation for CPU plugin: 41145cf - Fuse vectorized MatMul experts into MOE for 3GEMMs and 2GEMMs pattern: ``` class ov::pass::VectorizedExpertsFusion : public ov::pass::GraphRewrite { public: OPENVINO_GRAPH_REWRITE_RTTI("VectorizedExpertsFusion"); VectorizedExpertsFusion() { add_matcher<ov::pass::FuseVectorizedMOE2GEMM>(); add_matcher<ov::pass::FuseVectorizedMOE3GEMM>(); } }; ``` - Add internal MOE op MOE internal op spec PR: - #32255 ## Preliminary requirements (offline transformations): - Patterns match MatMul (transpose_a=False, transpose_b=**True**), for batched MatMuls preliminary update of MatMulConstTransposesExtraction is needed: - #32378 - Fusion of separate MatMul experts into vectorized (batched) MatMul: - #32199 ### Tickets: - transformation (and fusion details): 173663, op: 171913
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details:
they will not appear in the converted model public IR
Describes MOE used in PR:
Tickets: